Research Question

Which are the most valued data science skills?

Importing data

Kaggle ML and Data Science Survey, 2017

The survey data included:

1. free-response items
2. multiple choice items

Agenda

A Day in a Data Scientist’s Life

What do Data Scientists Want to Learn?

Data Science Methods

Learners vs. Employed Data Scientists

R vs. Python

Salary Comparison for Python vs. R

Conclusion

A Day in a Data Scientist’s Life

Demographics Information

Variables: Country, GenderSelect & Age

Demographics Insight:

1. Number of Respondents - 16716
   
2. Country - 50+
3. Gender - 13610 males (84%), 2778 females (16%).
4. Median Age - 30

A Day in a Data Scientist’s Life

Exploratory Data Analysis (EDA) - Demographics

A Day in a Data Scientist’s Life

Let’s take a peek at a day in the life of a Data Scientist and try to figure out what a data scientist does.

A Day in a Data Scientist’s Life

Manipulating data - DS Activities

Key Attributes: TimeGatheringData, TimeModelBuilding, TimeProduction, TimeVisualizing, and TimeFindingInsights.

Exploratory Data Analysis (EDA)

A Day in a Data Scientist’s Life

Manipulating data - Learning Platforms and Learners Sentiments

Key Attributes: 18 Learning Platforms (Arxiv,Blogs,College,Company,Conferences,Friends,Kaggle,Newsletters,Communities,Documentation,Courses,Projects,Podcasts,SO,Textbook,TradeBook,Tutoring,YouTube)

Exploratory Data Analysis (EDA)

What do Data Scientists Want to Learn?

Does survey takers’ formal education have any relationship to the Machine Learning/Data Science method he or she is most excited about learning in the next year?

Understanding what working professionals want to learn could give us insight into what skills are most valued in the field.

What do Data Scientists Want to Learn?

Variables and their definition

1. FormalEducation: Which level of formal education have you attained?
2. MLMethodNextYearSelect : Which Machine Learning/Data Science method are you most excited about learning in the next year?

These questions were asked to all participants.

What do Data Scientists Want to Learn?

Exploratory Data Analysis (EDA)

Mostly Master’s degrees followed by Bachelor’s then doctoral degrees.

What do Data Scientists Want to Learn?

Let’s look at the different Machine Learning/Data Science methods.

What do Data Scientists Want to Learn?

Distribution of Machine Learning/Data Science methods with formal education.

What do Data Scientists Want to Learn?

Data Science Methods

Overview

• Goal: further explore the data science methods used on the job by those writing code
• Methods used by data scientist can be equated in many ways to skills

Data Science Methods

Variables and their definition

• Response options for each method were: Rarely, Sometimes, Often, Most of the time
• Also used formal educational attainment variable

Data Science Methods

Manipulating data

• Filter to those writing code
• Filter to those who endorsed at least one data science skill
• Question was framed as: “At work, which data science methods do you use? (Select all that apply)”
• Top five most frequent data science methods endorsed were then selected and given a frequency score
• Grouping by formal education level attainment to identify most common data science methods

Data Science Methods

Exploratory Data Analysis (EDA)

• Total in sample: 7,773
• Nearly 2/3 of respondents endorsed data visualization
• Over half endorsed cross-validation and decision trees

Data Science Methods

• The following plot graphically displays the frequency of endorsements for the data science methods asked about.

Data Science Methods

• “Frequency Score” for the Top Five most endorsed data science methods
• Just because a method is endorsed, doesn’t mean that individuals use it frequently
• Converted responses to numeric values (Rarely = 1; Sometimes = 2, Often = 3, and Most of the time = 4)
• Data visualization used most frequently

Data Science Methods

• Frequency of methods endorsed for each formal education level assessed by Kaggle.

Data visualization remains the most frequently endorsed data science method.

Data Science Methods

The same data in tabular format:

Data Science Methods

• Most frequently endorsed: data visualization, logistic regression, cross-validation, decision trees, and random forests
• Also used quite frequently
• Unrelated to formal educational attainment
• Implications for students of data science
• Consider data visualization and the other aforementioned skills as critical

Learners vs. Employed Data Scientists

Is there a difference between what Learners think are the important skills to learn…

…and what employed Data Scientists say are the skills and tools they use?

Learners vs. Employed Data Scientists

Variables and Manipulating Data - Likert Scales

105 variables in 4 likert scale categories

• Learning Platform Usefulness - learners and data scientists
  • Not Useful   Somewhat useful   Very useful
• Job Skill Importance - learners
  • Unnecessary   Nice to have   Necessary
• Work Tools Frequency - data scientists
  • Rarely   Sometimes   Often   Most of the time
• WorkMethodsFrequency - data scientists
  • Rarely   Sometimes   Often   Most of the time

Plus a few basic demographic fields

Learners vs. Employed Data Scientists

Employed Data Scientists - Demographics

30% go by “Data Scientist”

20% go by “Scientist/Researcher” or “Software Developer/Software Engineer”

Learners vs. Employed Data Scientists

Learners - Demographics

About 73% are college students

55% are “focused on learning mostly data science skills” regardless of academic status.

Learners vs. Employed Data Scientists

Learning Platform Usefulness - Learners

Learners - Top 3 ways to learn data science:

1. Courses
2. Projects
3. College

Friends are the least useful learning platform.

Learners vs. Employed Data Scientists

Learning Platform Usefulness - Employed Data Scientists

• Projects and Courses belong in the top 3, but College is in 5th place
• Much greater importance on Friends…
• 46.5% say Friends are “Very useful”
• 97.5% say Friends are “Somewhat useful” or “Very useful”

Learners vs. Employed Data Scientists

Job Skills Importance to Learners

• 63.6% say Python is “Necessary”
• 39% say Data Visualization is “Necessary”
• 35% say R is “Necessary”

Learners vs. Employed Data Scientists

Work Tools Frequency

• 75.4% use Python either “Often” or “Most of the time”
• 63.6% use R either “Often” or “Most of the time”

Learners vs. Employed Data Scientists

Work Methods Frequency

Data Visualization is at the top of the list.

• 91% use it “Often” or “Most of the time”
• 0% use it “Rarely”

Only 38.8% of Learners said Visualizations were a “Necessary” job skill to learn!

R vs. Python

The most frequently used of the programming languages are R and Python. But do those that use R recommend R or Python? And do those that use Python recommend R or Python? In other words, do those survey takers feel that others should first and foremost study the languages they themselves have taken up, or perhaps with their insight, know to suggest the language of the two they themselves did not learn?

Thus the following questions were explored:

1. What is the distribution of following programming languages Kaggle survey takers used in the past year:

   • R Only
   • Python only
   • Both Python and R
   • Neither Python nor R

2. What is the distribution of programming language recommendations by following programming languages Kaggle survey takers used in the past year:

   • Using R Only
   • Using Python only
   • Using Both Python and R
   • Using Neither Python nor R

R vs. Python

Variables and their definition

There are 2 variables used in this section of the analysis :

1. LanguageRecommendationSelect=(What programming language would you recommend a new data scientist learn first? (Select one option) - Selected Choice)

2. WorkToolsSelect= For work, which data science/analytics tools, technologies, and languages have you used in the past year? (Select all that apply) - Selected Choice

R vs. Python

Exploratory Data Analysis (EDA)

## [1] 16716   229

R vs. Python

## [1] 7955  229

R vs. Python

R vs. Python

R vs. Python

Let’s examine the above graph of LanguageRecommendationSelect

R vs. Python

## [1] 7955    5

R vs. Python

Salary Comparison for Python vs. R

Finally, true to the word “value,” considerations have to be made regarding pay. The compensation received by survey takers for their work in either R or Python needs quantification to discover which language earns a data scientist more overall and in general.

Salary Comparison for Python vs. R

Contributing Variables

Three variables were used:

• WorkToolsSelect: a “select all that apply” variable with a list of various data science tools, technologies, and languages
• CompensationAmount: a numerical value to indicate their annual pay
• CompensationCurrency: a character string that indicated currency of annual pay

There was also the variable “id” that was created for the purpose of this report, acting as a way to identify each individual survey taker, and the variable “work_tools” which was a derivative of WorkToolsSelect, breaking the lists down into their individual components.

Salary Comparison for Python vs. R

Exploration and Review of Compensations

Conclusion

On the contentious debate on which Machine Learning/Data Science methods Data Scientists are most excited about learning in the next year as the most valued Data Science Skills

• Deep Learning is the top most Machine Learning/Data Science method in all categories of formal education followed by Neural Nets except High school graduates, all others wants to learn
• Time Series Analysis as the third Machine Learning/Data Science method. High school graduates want to learn Genetic & Evolutionary Algorithms as their third choice.
• Among doctoral survey takers, Bayesian Methods is the third preference.

Conclusion

On Data Science Methods Used on the Job

• Data Visualization is a remarkably popular data science method. It is the most endorsed by nearly all education attainment levels.
• Cross validation, random forests, logistic regression, and decision trees are also heavily endorsed.
• These are not just short but required or essential tasks–not only do so many of those writing code use data visualization, but they also use it quite frequently
• The data suggest that data science methods do not differ much between formal educational attainment groups.

Conclusion

Learners vs. Employed Data Scientists

• Both Learners and employed Data Scientists agree that Courses, Projects and College are in the top three ways to learn Data Science
• Learners place much less importance on Friends for learning than employed Data Scientists do
• Learners place a higher importance on Python vs. R as compared with employed Data Scientists
• Data Visualization is the top used skill for working Data Scientists even though learners put relatively little importance on it as a Job Skill

Conclusion

On Data Science Activities

• Gathering Data is the main activity where data scientists spend most of their time followed by model building.
• Personal projects and Online Courses appear to be very useful learning platforms.

Conclusion

On the contentious debate on R vs Python as the most valued Data Science Skills

• Half of the sample uses both R and Python
  
• R only to Python only users are in 1:2 ratio
  
• More R users recommended Python than the Python users recommended R
  
• Both users recommendations in Python is more than their recommendation in R
  
• R users are more likely to have a higher base salary, but Python users have the greater potential for wage growth